Appendix G — Practice Final Solutions

STAT303-2 (Winter2023)

Abstract

These solutions are being composed by students of the course STAT303-2 (Winter 2023).

G.1 Potential problems

Presence of which of the following potential problems in a linear regression model may lead to statistically significant variables appearing insignificant?

1. Multicollinearity

2. Outliers

3. Overfitting

Answer: A and B

Explanation:

A) Multicollinearity:

Recall, the estimated variance of the coefficient \(\beta_j\), of the \(j^{th}\) predictor \(X_j\), can be expressed as:

\[\hat{var}(\hat{\beta_j}) = \frac{(\hat{\sigma})^2}{(n-1)\hat{var}({X_j})}.\frac{1}{1-R^2_{X_j|X_{-j}}}\]

If the predictor \(X_j\) is collinear with other predictors, \(R^2_{X_j|X_{-j}}\) will be large, which in turn will inflate \(\hat{var}(\hat{\beta_j})\). In other words, multicollinearity inflates the standard errors of the coefficients for which the variables are collinear. Since \(t\)-statistic is calculated by dividing the estimated coefficient by its standard error, the \(t\)-statistics shrinks, and the corresponding \(p\)-value increases. Therefore, the hypothesis test loses the power to reject the null hypotheses, and thus statistically significant variables appearing insignificant.

Another way to think about this can be that if some predictors are collinear, it can be difficult to separate out the individual effects of these variables in the response and significant variables may appear insignificant.

B) Outliers

Recall, the estimate of error variance is given by:

\[\hat{\sigma}^2 = {\frac{RSS}{n-2}},\] where RSS is the residual sum of squared errors. Outliers result in an increase in \(RSS\), leading to an increase in the estimated error variance \(\hat{\sigma}^2\), which in turn inflates \(\hat{var}(\hat{\beta_j})\). The rest of the explanation follows from the previous explanation on multicollinearity.

C) Overfitting

Overfitting shrinks \(RSS\), which in turn shrinks \(\hat{\sigma}^2\), thereby shrinking \(\hat{var}(\hat{\beta_j})\). Thus overfitting will act in way opposite to what we observe in (A) and (B).

G.2 Potential problems

Classify a data point as influential / outlier / high leverage in a linear regression model, based on the description.

1. The data point is likely to have a large effect on the model in terms of prediction: Influential point

2. The data point has the potential to have a large effect on the model in terms of prediction: High leverage point

3. The data point is likely to inflate the model R-squared: High leverage point that is not influencial

4. The data point is unlikely to have a large effect on the model in terms of prediction: outlier

Explanation:

See the graphics in class presentation on Chapter3_Outliers_high_leverage_influential_points. Think of influential points / high leverage points / outliers as a force (proportional to the residual corresponding to the point) pulling a canteliver beam. Depending on the position from where you pull the cantilever beam, you may move it too much or too little.

A) Inluential point (high leverage & outlier): an outlier with the respect to both the predictor and the response. It has a large effect on the regression line. As shown in the graphics, influence is higher for more extreme outliers with same leverage and for points with higher leverage & similar outlying distance.

B) High leverage point: Observations with high leverage have an unusual value for the predictor (ie. lie outside the domain of most points). High leverage point has the potential to have a large affect on the regression line. It is cause for concern if the least squares line is heavily affected by just a couple of observations, because any problems with these points may invalidate the entire fit.

C) If you have a high leverage point that is not influencial: The variance of the response may increase in the presence of high leverage points, since an unusual set of predictor values may correspond to an unusual response, which may increase the total variation. However, as the point is not inluential, the increase in the unexplained variation (the squared residual) will not be proportionate to the increase in total variation. As \(R^2\) is one minus the ratio of unexplained variation to total variation, it is likely to increase.

D) Outliers: As shown in the graphics, outliers very small effect on prediction.

G.3 Autocorrelation

A linear regression model was developed to predict the number of passengers taking a flight per month. The data consists of number of passengers flying each month from January 1949 to December 1960. The autocorrelation plot below shows the correlation of the residuals with the lagged residuals of the model. Choose the most appropriate option.

1. The above plot shows the presence of autocorrelation. The 6-month lagged response is the most appropriate lag to be added as a predictor in the model to address autocorrelation

2. The above plot shows the presence of autocorrelation. The 12-month lagged response is the most appropriate lag to be added as a predictor in the model to address autocorrelation

3. The above plot shows the presence of autocorrelation. The 1-month lagged response is the most appropriate lag to be added as a predictor in the model to address autocorrelation

4. The above plot shows the absence of autocorrelation as the plot must have a cyclical pattern in the presence of autocorrelation

5. The above plot shows the absence of autocorrelation as the one month lagged residual must have the highest correlation with the residual in the presence of autocorrelation

Answer: B

Explanation: As seen in the plot, the residuals are highly correlated (correlation of more than 60%) with lagged residuals of 12 months. This shows the presence of autocorrelation. To address autocorrelation, the 12-month laggged response will be the most appropriate as it has the highest correlation with the response. Thus, it will explain the variation in the respone the most.

There is no need for there to be a cyclical pattern for autocorrelation. Even if one of the lagged residuals are highly correlated with the residual, it shows the presence of autocorrelation.

G.4 Logistic regression (goodness-of-fit)

Which of the following metrics can be used to assess the goodness-of-fit of a logistic regression model?

1. All of these

2. LL-Null

3. Log-Likelihood

4. Df Model

5. R-squared

Answer: Log-Likelihood

Explanation In logistic regression, the response is assumed to follow a Bernoulli distribution, where the probability of success is a function of the predictors and its coefficients (the model parameters). With this assumption, one can compute the the joint probability density of the observed data as a function of the model parameters. This creates a set of probability distributions (based on different values of model parameters) that could have generated the data. The algorithm finds the values of the model parameters (the beta coefficients) such that the probability of observing the data maximizes. This probability is the likelihood, and its logarithm is the log-likelihood. The higher the log-likelihood, the more probable it is to observe the data. Thus, log-likelihood is a way to measure the goodness-of-fit of the model.

LL-NULL is the log-likelihood of the model with no parameters. This is compared with the log-likelihood of the model with predictors to test if the regression is statistically significant.

Df Model is the number of predictors in the model.

R-squared cannot be used for logistic regression as there are no residuals.

G.5 Logistic regression (threshold probability)

For a logistic regression model, as we increase the decision threshold probability,

1. None of these

2. the recall will reduce or stay the same

3. the ROC-AUC will increase or stay the same

4. the precision will increase or stay the same

5. the classification accuracy will increase or stay the same

Answer: B

Explanation: See class slide on the confusion matrix below.

Increasing threshold probability means that less observations are predicted to be positive. Hence, some TP could turn into FN, reducing the recall. (this might not happen if there is no observations of actual positives between the thresholds). ROC-AUC is independent of the threshold probability. Both precision and classification accuracy might decrease if the number of FP among actual negatives increase more than the increase of TP among actual positives by the shift in the threshold.

G.6 Decision threshold probability

Which of the following metrics is independent of the decision threshold probability?

1. None of these

2. ROC-AUC

3. All of these (except the “None of these” option)

4. Precision

5. Recall

Answer: ROC-AUC

Explanation By changing the threshold, the number of points classified as negative and positive may change, and so TP, FP, TN and FN may change. Recall and precision may change as they are based on these metrics (TP ,FP, TN, and FN). However, the ROC-AUC specifically analyzes different thresholds. The ROC curve is a plot of TPR against FPR for all possible thresholds, and ROC-AUC is the area under the ROC curve, so the value itself is independent from the decision threshold probability.

G.7 Odds

Consider the following logistic regression model:

\[p(x) =\frac{1}{1+e^{-(\beta_0+\beta_1x)}}\].

Which of the following metrics will depend on the value of x?

1. Odds ratio when x increases by 2 units

2. increase in log odds when x increases by 10 units

3. All of these

4. Increase in predicted probability when x increases by 1 unit

E)none of these

Answer: D

Explanation:

\[p(x) =\frac{1}{1+e^{-(\beta_0+\beta_1x)}}\]

\[\implies \log\bigg(\frac{p(x)}{1-p(x)}\bigg) = \beta_0 + \beta_1x\]

\[\implies \log\big(Odds(x)\big) = \beta_0 + \beta_1x\]

When \(x\) increases by ‘c’ units,

\[p(x+c) - p(x) =\frac{1}{1+e^{-(\beta_0+\beta_1(x+c))}}-\frac{1}{1+e^{-(\beta_0+\beta_1(x))}}\]

\[log({Odds(x+c)}) - log({Odds(x)}) = \beta_1 c\]

\[\frac{Odds(x+c))}{Odds(x)} = e^{\beta_1 c}\]

We can see that only the increase in predicted probability when \(x\) increases by 1 unit is dependent on \(x\).

G.8 Precision-recall

We develop a logistic regression model to predict whether someone will pay a loan back or not. Loans are “approved” by us only for those borrowers who are predicted to pay back. The positive class is the borrowers that pay back the loans. What would a recall of 81% mean?

1. 81% of the borrowers that would pay back the loan are approved by us: Recall = TP/(TP + FN). TP here are those who are [approved by us] who [pay back the loan], while FN are those who were [not approved by us] but actually [pay back the loans]. The denominator is [all who pay back the loan]. Thus, Recall here means: among [all who pay back the loan], 81% are [approved by us].

2. Of all the loans we approve, 81% pay us back: This is Precision = TP/(TP + FP)

3. Of all the loans we don’t approve, 81% would not have paid us back if they were given the loan: This is the proportion of negatives correctly predicted - like precision for the negative class

4. Of all the loans we don’t approve, 19% would not have paid us back if they were given the loan: This is the proportion of negatives incorrectly predicted.

Answer: 81% of the borrowers that would pay back the loan are approved by us.

Explanation: Recall = True Positives/(True Positives + False Negatives).

In this case, True positives are those who got approved and would pay back. False Negatives are those we didn’t approve, but would pay back. Therefore, 81% Recall means 81% of the borrowers that would pay back the loan are approved by us.

G.9 Variable selection

Which of the following algorithms can be used for variable selection?

1. Lasso

2. Ridge regression

3. Forward stepwise selection

4. Best subset selection

Answer: A,C,D

Explanation: Both lasso and ridge regression are regularized least squares model, where the a shrinkage penalty is added to the ordinary least squares cost function. The shrinkage penalty in ridge regression shrinks the regression coefficients estimate towards zero, but not exactly zero, while the shrinkage penalty in lasso tends to give a set of zero regression coefficients and leads to a sparse model. Therefore, lasso can be used for variable selection, but not ridge regression.

Forward stepwise and best subset selection are variable selection algorithms by fitting multiple models having different combinations/number of predictors and choosing the best model.

G.10 Precision-recall

You are building a facial recognition model to allow people to unlock their phone. If the phone recognizes the person as the authorized user, it will unlock the phone. If it doesn’t recognize the user, it will prompt them to try again or try an alternative method (such as a passphrase). The facial recognition model is a classification model that identifies if the person unlocking the phone is the authorized user (positive response) or not (negative response).

Assume that letting a stranger (unauthorized user) unlock the phone is more risky (or more expensive) than not letting the authorized user unlock the phone.

Which of the following metric is the most important to optimize in the model?

1. Precision

2. Classification accuracy

3. Recall

4. ROC-AUC

Answer: Precision

Explanation:

A) Precision: Precision = TP/(TP + FN). Here, FN are those who are falsely assigned as an unauthorized user when they are actually the authorized user. FP are those who are assigned as the authorized user and are actually an unauthorized user. In this case, it’s important to optimize precision because it is more important to reduce the number of FP (strangers being recognized as the authorized user) than to reduce the number of FN (authorized user not being recognized).

B) Classification accuracy: This is incorrect because a model with high accuracy but a high FPR would be unacceptable since it would increase the risk of a stranger unlocking the phone.

C) Recall: This is incorrect because a high recall indicates that many of the positive cases are being detected. However, it does not measure the fraction of unauthorized users that the model identifies as authorized. A high FPR could lead to an unauthorized user unlocking the phone, which is a more expensive mistake than an FN.

D) ROC-AUC: This is incorrect because ROC-AUC does not take into account the cost of the positive and negative classes. It only measures how well the model can distinguish between authorized users and unauthorized users.